Scoring Module Dev: atomic stage-pipeline protocols with official-result validation - #149

judge_include_reason changes the judge prompt ("Include a concise reason" vs "Set reason to an empty string") and judge_max_tokens changes truncation/parse-failure behavior — both affect verdicts but are missing from the resume fingerprint, so resuming across a toggle silently mixes verdicts from two judge configs. Please add both to this LLM branch.

Done, both are now in the LLM branch of the fingerprint.

Asymmetry: compact "AC" is accepted for predictions here, but parse_gt_letters rejects compact gts — so a dataset storing gt as "AC" scores 0 forever, silently. Suggest making it symmetric by NOT accepting compact form anywhere (it's uncommon and ambiguity-prone) and leaving compact-format datasets to a user patch.

Rather than dropping compact parsing (it's a real output format), we made it symmetric where it's unambiguous: matchers now accept compact gts once the row is typed mcq, restricted to the sample's actual candidate letters. Type inference alone still refuses compact form since "BED"-style words would misclassify.

accuracy divides by total including invalid samples, which deflates scores on answer-withheld/malformed splits. Please emit both accuracy and accuracy_valid (correct/valid) with a 1-line comment explaining the difference.

Done, summary now carries both: accuracy (over all rows, invalid counts as wrong) and accuracy_valid (over gradable rows only).

When gt == 0 this turns rel_tol into an absolute tolerance (pred 0.04 passes at rel_tol 0.05). ChartQA's official relaxed accuracy falls back to exact equality at target 0 — please exact-match when gt==0 (with abs_tol still honored), otherwise the official_protocol stamp is inaccurate for these rows.

Fixed, numbers_match no longer applies numeric_rel_tol at gt == 0 (abs_tol still honored), matching the official implementation (lmms-eval chartqa relaxed_correctness: the target_float truthiness check
routes target==0 to exact match).

If tmp passes the fingerprint we return only tmp's rows, dropping a fuller same-fingerprint final score.json. Union (final first, tmp overwrites) recovers more resumable rows — though that may require synced changes elsewhere (flush/finalize logic), so fine as a follow-up.

Fixed, final score.json loads first, tmp overlays it, both fingerprint-gated. A completed run also always clears its tmp now, so a stale tmp can never overlay a fresher final.

qwenvl2d5_my (plus *-merged now living in two series) looks like a personal experiment series leaking into the shared registry — and makes get_series order-dependent. Drop or rename before merge.

Dropped.

Docstring says the loader "ensures grading fields sit at the sample top level", but _process_sample never lifts any fields — the scorer rejects old-layout files instead. Please fix the docstring to match.

Fixed. Docstring now states the actual contract: rows are expected flat; non-conforming rows pass through unchanged and surface at scoring time via the scorer's explicit layout rejection.

All four regenerated fixtures are 100% invalid: MMBench 20/20, hf_template 20/20, LLaVABench 60/60, modality_test 18/18 — MMBench's answer is '' (answer-withheld split) and the LLaVABench rows lack reference_response. So the fixtures exercise only the invalid-sample path: zero matcher/grader/judge code is covered, and they'd green-light a fully broken scorer. Please regenerate from a split with answers (e.g. MMBench dev) and add an invalid == 0 assertion to the fixture-generation step.

Fixed as you suggested.

Note this goes the opposite direction from my suggestion (reject compact everywhere) — accepting compact gts at match time is a defensible alternative, but it has a residual collision: with the A–F fallback candidates (options-in-image datasets), word-like gts parse as letter sets — "BED" → (B,D,E), "FACE" → (A,C,E,F). Please guard it: only accept compact gts when the candidates come from real parsed options, not the A–F fallback (same for template.py / llm.py) — or revert to rejecting compact form.

Fixed as you suggested. Compact gt parsing now requires the candidates to come from real parsed options; under the A–F fallback a word-like gt ("BED", "FACE") is treated as a whole-string answer instead of a letter set.

This fixture is still 60/60 invalid (and modality_test is 18/18) — the round-2 regeneration covered only the two MMBench fixtures, and the invalid == 0 gate lives only in those two scripts. LLaVABench is the judge-path example, so the llm-judge stage still has zero fixture coverage; everything here exercises only the missing-gt path. Please regenerate it with reference_response present (dev rows) and extend the fixture gate to this script — or drop the degenerate score.json fixtures (LLaVABench, modality_test) rather than committing all-invalid examples.

Regenerated.

Root cause: LLaVABench keeps its reference answer in gpt4_ans, not the default answer field, so gt resolved to None and every row fell to missing_gt_or_pred; with no dataset_meta the default exact-match,rule-match also ran in place of the declared llm-judge protocol.

Re-scored with --score_gt_field gpt4_ans --score_pipeline llm-judge under an open-weight local judge (Qwen2.5-7B-Instruct, greedy): 60/60 valid, 0 invalid, 0 llm_errors, all 60 rows decided by llm-judge.

The resume fingerprint covers every config knob but carries no dataset identity — and the cache's only binding to its result.json is "same folder". Same-dataset variants collide silently: rerun the same benchmark into the same out_dir with --split test after a dev run (or --subset cn after en, or after the dataset is re-pushed upstream with corrected rows) and the fingerprint passes, eval-ids 0..N collide, and the old split's verdicts are stamped onto the new rows with no trace.

Please have the loader inject dataset_name / subset / split into dataset_meta (it already knows all three) and include them here — that also makes score.json self-describing about what it scored, which VALIDATION.md's table currently asserts from outside the artifact.

(Doesn't cover same-dataset re-inference with a new checkpoint — a per-row sha256(pred) check would; fine as a follow-up.)

Done. The loader now injects dataset_name/subset/split into dataset_meta (mmeval_hf.py). A rerun of a different split/subset/dataset into the same out_dir is now a cache miss and is re-scored instead of inheriting stale verdicts.

Stale remediation: "the current loader translates published metadata to score_pipeline" is no longer true — the translation boundary was deleted this commit, and the loader now hard-fails on score_type with its own retirement error (mmeval_hf.py). A user following this message re-runs inference and dead-ends on the loader's error, each message pointing at the other component. Please change the remediation here (and keep the loader message consistent) to: "re-push the dataset with the score_pipeline schema, then re-run inference."

Fixed. Neither message references the old format's history anymore; they state that the score_type key is unsupported and point to the score_pipeline schema (docs/en/SCORING.md, 'Dataset metadata contract'), then re-run inference.

This PR's history includes b0b89a8 ("eval: render MCQ options for HRBench/MMVP"), which added ~15 lines right here in build_prompt: rebuild options from flat per-letter keys (HR-Bench A/B/C/D) or a single options_text string (MMVP) so the template's {% if options %} actually fires. Per that commit's own message, without it "the model never sees the options and scores collapse to ~random" — it was validated as part of this branch's parity work.

The sync-merge cd1fb16 resolved the base.py conflict by taking main's side, and the hunk is gone from the head tree (no options_text anywhere in mmeval/data/) — so the PR's commit log advertises a fix its diff no longer contains. Anyone reading merged history will believe it landed.

Was dropping it intentional? If deferred as out-of-scope for the scoring PR, please say so in the PR description and file a follow-up to restore it; if not, cherry-pick the hunk back from b0b89a8. Either is fine — the current silent state is the only bad option.

It is intentional. Option normalization was moved out of build_prompt and into the dataset loader part: datasets can declare their own prompt template in metadata.json, and the TSV loader rebuilds the options dict itself, which makes a generic rebuild inside build_prompt redundant.

Both datasets are still covered, and I verified the affected paths end-to-end: mm-eval/MMVP renders through its own "{{ options.options_text }}" template; evalkit's MMVP.tsv is index/image/question/A/B/answer, so tsv.py:158 rebuilds the options dict for the default template; HRBench4K is covered the same way on both paths.

Restoring the hunk would only add dead code: its guards fire only when options is empty and the choices sit in top-level keys, and neither holds in the current sample shape.